Apparatus and method for image display

ABSTRACT

A mobile data apparatus equipped with a display is improved in both operability and mobility include a frame difference detector  4  for detecting a change in image between two consecutive frames transferred from a pair of frame buffers  2  and  3 . The change in image is used in a barycenter calculator  5  for calculating the barycenter of each image. The barycenter is saved in a barycenter coordinates memory  6 . A camera movement calculator  6  calculates a difference between the barycenter saved in the barycenter coordinates memory  6  and a reference barycenter saved in a barycenter coordinates memory  7  to determine a horizontal pixel shift DPx and a vertical pixel shift DPy of a camera. In response to those shifts, a horizontal angle-of-rotation controller  9  and a vertical angle-of-rotation controller  10  determine angles of rotation along the horizontal and vertical directions hx and hy for a virtual camera  11 . Accordingly, the view of the virtual camera  11  can duly be controlled.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method fordisplaying an image of view point information produced with aremote-controlled camera.

2. Description of Related Art

As a wide range of Internet services have overwhelmingly been provided,their displaying data include not only texts, still images, and soundsbut also motion images and 3-D dynamic model spaces. For developing such3-D dynamic model spaces, the VRML (virtual reality modeling language)was standardized in August, 1996 and used commonly as a today's standardtechnology. This technology allows a variety of 3-D model spaces of oneformat, which may simulate museums and theaters for example, to beconstructed as information to the Internet and accessed by any user whooperates its computer connected to the Internet to enjoy as if he or shewalks actually around in a museum or a theater. For that purpose, animage displayed on a display unit of the computer is based on the viewof a virtual camera installed virtually in the 3-D model spaces. Uponreceiving a command from the user requesting walk about in a 3-D modelspace, the computer controls the virtual camera for desired positioningand orientation. As a result, an image is displayed on the display ofthe computer showing as if the user walks about in the 3-D model space.

The command to the computer may be given by the user operating with itshand or fingers a pointing device, such as a mouse or a joystick, or atouch display in all directions, such as upward, downward, leftward, andrightward.

The commanding of the computer with the use of a hand or fingersoperating the pointing device, such as a mouse or a joystick, or thetouch display in all upward, downward, leftward, and rightwarddirections is substantially suited for providing a modification or acommand on the screen displaying an image having relatively less motion,such as a text or a still image. It is however unfavorable to controlquick movements of a view point or the view of the virtual camera in 3-Dmode spaces. It is routine that when a person changes its view point indaily life, its eyes or head is shifted to a desired location but rarelymoves and orients the object to be viewed.

This drawback may not be critical with a desk-top type the computerwhere the view of the user is fixed to its display screen at astationary location while controlling an indicator or pointer on thescreen by manipulating the control with the hand or fingers.

However, when a specific type or namely, a head-mounting type of thedisplay is used, it is hung on the front of the eyes of the user andmoves together with the head. When a hand-held type of the display isused, it is held by the hand of the user and turned to any of upward,downward, leftward, and rightward directions in response to the movementof the head so that it can stay at the front of the user's eyes.

In a system equipped with a mobile type of the display which can be heldin the front of the eyes of a user, it is disadvantageous fordynamically shifting through a computer the view point of a virtualcamera in a 3-D model space to use a pointing device, such as a mouse ora joystick, or a touch device which can be controlled by the hand orfingers in upward, downward, leftward, and rightward directions.

It is also known in a virtual reality technology or a tele-existencetechnology for controlling the dynamic movement of a view point with acomputer that the view point of a user is normally measured by ameasuring device, such as an eye-tracking device or a heat-positiontracing device, which can be fitted to a body of the user. The fittingof the device to the body will add another inconvenience to the user,impairing the operability and the mobility.

SUMMARY OF THE INVENTION

It is a primary object of the present invention to provide an apparatusand a method for displaying an image where the view point of a virtualcamera can dynamically be shifted in response to the movement of amobile set equipped with a display.

It is another object of the present invention to provide an apparatusand a method for displaying an image where the mobile set equipped witha display is improved in both the operability and the mobility.

An image displaying apparatus according to the present inventionincludes a controller which processes input image data produced by animage input unit which pictures the background of an image display unitin order to detect a movement of the image display unit, and an imagedata generator which generates data indicative of the view of a camerathat is remote controlled in response to the detected movement, wherebythe image data can be displayed on the image display unit.

An image displaying method according to the present invention comprisesthe steps of picturing the background of an image display unit,processing pictured data to detect a movement of the image display unit,generating image data of the view of a camera that is remote controlledin response to the detected movement, and displaying it on the imagedisplay unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a functional arrangement of an imagedisplaying apparatus according to the present invention;

FIG. 2 is a flowchart showing actions of the image displaying apparatus;

FIG. 3 is a front view of a mobile data assistant associated with thepresent invention;

FIG. 4 is a side view of the mobile data assistant.

DESCRIPTION OF PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described in moredetail referring to the accompanying drawings.

An image displaying apparatus according to the present invention has anarrangement shown in FIG. 1. The image displaying apparatus is a mobiletype image displaying apparatus comprising a microcomputer 14, and animage input unit 1 and an image display unit 13 both connected to themicrocomputer 14.

The image input unit 1 comprises a CCD camera for picturing thebackground of the image display unit 13. The image display unit 13 is aliquid crystal display for displaying an image of visual informationsupplied from the microcomputer 14.

The microcomputer 14 includes, as shown in the functional block diagramof FIG. 1, a frame difference detector 4 to which image data generatedby the image input unit 1 is supplied via frame buffers 2 and 3, abarycenter calculator 5 connected to the frame difference detector 4, acamera movement calculator 8 connected via barycenter coordinatesmemories 6 and 7 to the barycenter calculator 5, a horizontalangle-of-rotation controller 9 and a vertical angle-of-rotationcontroller 10 both connected to the camera movement calculator 8, avirtual camera 11 connected to the horizontal and verticalangle-of-rotation controllers 9, 10, and a 3-D object space database 12connected to the virtual camera 11 for constructing a 3-D model space.

The image displaying apparatus performs a succession of actions, asshown in a flowchart of FIG. 2, including moving its body equipped withthe image display unit 13 (Step S1), saving in the frame buffer 2 eachframe of input image data generated by the image input unit 1 imagingthe background of the input display unit 13, and transferring the saveddata of the frame buffer 2 to the frame buffer 3 before the succeedingframe is fed to the frame buffer 2 (Step S2), and saving the succeedingframe of the image data (Step S3).

The frame difference detector 4 calculates an optical flow between twoconsecutive frames from the saved data of the frame buffers 2 and 3 todetect a change in the image between the two frames. The change in theimage is transferred to the barycenter calculator 5 where it is used tocalculate the barycenter of the change which is then saved in thebarycenter coordinates memory 6. In the other barycenter coordinatesmemory 7, the barycenter is initialized to (0, 0).

The camera movement calculator 8 calculates a difference between twooutputs of the barycenter coordinates memories 6 and 7 to determine ahorizontal pixel level shift ΔPx and a vertical pixel level shift ΔPy(Step S4). Then, the saved data of the barycenter coordinates memory 6is transferred to the barycenter coordinates memory 7.

This is followed by the horizontal angle of rotation controller 9 andthe vertical angle of rotation controller 10 calculating an angle ofrotation along the horizontal direction θx and an angle of rotationalong the vertical direction θy respectively from the resultant shiftsof the camera movement calculator 8 to control the point of view of thevirtual camera 11 (Step S5).

The virtual camera 11 acquires object data in the 3-D model space fromthe 3-D object space database 12 and projects 3-D objects on the twodimensional screen as seen in its view. A resultant 3-D image isdisplayed on the liquid crystal display screen of the image display unit13 (Step S6).

The angle of rotation along the horizontal direction θx and the angle ofrotation along the vertical direction θy for controlling the virtualcamera 11 at Step S5 may be calculated from the following equations. Itis assumed herein that the CCD camera of the image input unit 1 isspaced substantially equally from subjects around by a distance d. Thenumber of pixels along the horizontal direction in one frame is Px andalong the vertical direction is Py. The value along the verticaldirection may be approximated from that along the horizontal directionand the following explanation is based on the horizontal direction. Whenthe CCD camera of the image input unit 1 is turned through the angle ofrotation along the horizontal direction θx, the actual distance ofmovement on the screen is Lx.

θx=tan⁻¹(Lx/d)  (1)

As the shift of the pixel level in one frame is ΔPx in response to themovement through θx, Lx_(MAX):Lx=Px:ΔPx is established where Lx_(MAX) isan actual distance of movement. Then,

Lx=(Δpx/Px)·Lx _(MAX)  (2)

When the equation (1) is then expressed by the equation (2),

θx=tan⁻¹(Lx _(MAX)/(Px·d))·ΔPx  (3)

There is now no variable. Similarly, along the vertical direction isexpressed by:

θy=tan⁻¹(Ly _(MAX)/(Py·d))·ΔPy  (4)

Accordingly, if Lx_(MAX)=d=80 cm and the number of pixels along thehorizontal direction Px is 160 at ΔPx=160, the camera is turned by 45degrees to the right.

The image displaying apparatus having the foregoing arrangement allowsthe horizontal angle-of-rotation controller 9 and the verticalangle-of-rotation controller 10 of the microcomputer 14 to detect amovement of the image display unit 13 through processing the input imagedata supplied from the image input unit 1 via the two frame buffers 2and 3 so that they function as a controller unit for remote-control ofthe camera view, and permits the virtual camera 11 and the 3-D objectspace database 12 to function in a combination as an image datagenerator for generating image data of the camera view remote controlledby the above mentioned controller unit. In operation, the image data ofthe camera view remote controlled in response to the motion of the imagedisplay unit 13 can be generated by the virtual camera 11 and displayedon the image display unit 13.

FIG. 3 is a front view of a mobile data assistant 100 associated withthe present invention. FIG. 4 is a side view of the mobile dataassistant 100.

The mobile data assistant 100 is a hand-held computer including amicrocomputer 14 of the image displaying apparatus shown in FIG. 1. Themobile data assistant 100 has a small-sized video camera connected by avideo cable 17 to a video capture board 16 and operated as the imageinput unit 1. Image data produced by the image input unit 1 imaging thebackground of the image display unit 13 equipped with a liquid crystaldisplay is fed to the microcomputer 14 as a data input for detecting amotion of the image display unit 13. Similar to the operation of theimage displaying apparatus shown in FIG. 1, the mobile data assistant100 permits the image data of the camera view remote controlled inresponse to the motion of the image display unit 13 to be generated bythe virtual camera 11 and displayed on the image display unit 13.

In the mobile data assistant 100, the image input unit 1 for imageprocessing is separated from a controller of the virtual camera 11 toproduce and supply only data of the angle of rotation along thehorizontal and vertical directions for the camera 11 thus contributingto the versatility of the entire arrangement. This permits anycombination of three different units: the image input unit 1 for imageprocessing, the image display unit 13, and the virtual camera 11 to becontrolled, depending upon the desired application.

The mobile data assistant 100 detects upward, downward, leftward, andrightward movements of a body of its image display unit 13 from imageprocessing and uses the data to shift the view of the virtual camera ina 3-D model space displayed on the screen of the image display unit 13.Accordingly, the mobile data assistant 100 permits upward, downward,leftward, and rightward views in the 3-D model space to be created ascloser as seen by the human eyes while its mobility and operability areimproved.

The present invention is not limited to the prescribed embodiments whereall the 3-D model space data is recorded in the 3-D object spacedatabase 12 provided as a fundamental function element of themicrocomputer 14 in the main body. Any type of the 3-D model space datacan be acquired from external sources. For example, the microcomputer 14may be communicated with the Internet through a modem port of a portabletelephone not shown to receive a desired type of the 3-D model spacedata from the Internet.

Also, the present invention is not limited to such a model as describedpreviously where the virtual camera 11 is virtually located in a 3-Dmodel space for implementing the image data produced by an image datagenerator of the microcomputer 14 or a combination of the camera 11 andthe 3-D object space database 12 and displayed on the image display unit13. The image data generator may include an actual video camera, notshown, which is located in a real space and can be remote controlled bythe controller for having a desired view.

Although the small-sized video camera of the image input unit 1 in themobile data assistant 100 is connected via the video capture board 16 tothe hand-held computer, it can be built integral with the hand-heldcomputer.

As set forth above, according to the present invention, the movement ofthe image display unit is detected through processing input image dataproduced by the image input unit picturing the background of the imagedisplay unit and used for generating and displaying the view of a cameraremote controlled on the image display unit. This permits the imagedisplay unit to be always kept in the front of the eyes of a userthrough shifting the apparatus in any of upward, downward, leftward, andrightward directions.

What is claimed is:
 1. An image displaying apparatus comprising: aportable image display unit; an image input unit mounted on the portableimage display unit for providing a base image corresponding to a view ofthe portable image display unit at an initial position; a controllerprocessing input image data produced by the image input unit fordetecting a movement of the view of the portable image display unit to asubsequent position and for remote controlling a view of a virtualcamera in accordance with a detected movement; an image data generatorfor generating data indicative of the view of the virtual camera remotecontrolled by the controller, in which a resultant image generated bythe image data generator is displayed on the portable image displayunit; and a movement calculator for calculating a horizontal pixel levelshift and a vertical pixel level shift of the portable image displayunit based on the detected movement of the view of the portable imagedisplay from the initial position to the subsequent position translatedto an angle of the rotation along a horizontal direction and an angle ofrotation along a vertical direction for controlling the view of thevirtual camera and the resultant display on the portable image displayunit.
 2. The image displaying apparatus according to claim 1, whereinthe image data generator comprises a virtual camera device forgenerating data indicative of the view of the camera remote controlledby the controller from object data in a 3-D model space, said objectdata being saved in and supplied from a 3-D object space database. 3.The image displaying apparatus according to claim 1, wherein the imagedata generator comprises a microcomputer provided with a communicationfunction so that the data indicative of the view of the camera remotecontrolled by the controller is generated from object data in a 3-Dmodel space obtained by the communication function.
 4. The imagedisplaying apparatus according to claim 1, wherein the image datagenerator comprises a camera device located in a real space and remotecontrolled by the controller for having a desired view.
 5. The imagedisplaying apparatus according to claim 1, wherein the controllerdetects the movement of the image display unit from a change in thebarycenter of the input image between two consecutive frames produced bythe image input unit.
 6. An image displaying method for a hand-heldportable image display unit comprising the steps of: providing a baseimage corresponding to a view of the portable image display unit at aninitial position by using an image input unit mounted on the portableimage display unit; processing image data to detect a movement of theview of the portable image display unit from the initial position to thesubsequent position; generating image data of the view of a virtualcamera remote controlled in response to the movement of the view of theimage input unit mounted on the portable image display unit detected insaid step of processing; and displaying on the portable image displayunit image data of the view of the virtual camera controlled in responseto the detected movement of the view of the portable image input unitfrom the initial position to the subsequent position.
 7. The imagedisplaying method according to claim 6 further comprising the step ofgenerating the image data of the view of the virtual camera remotecontrolled in response to the detected movement of the image input unitfrom object data in a 3-D model space saved in and supplied from 3-Dobject space database.
 8. The image displaying method according to claim6 further comprising the step of generating the image data of the viewof the virtual camera remote controlled in response to the detectedmovement of the image input unit from object data in a 3-D model spaceobtained by the communication function of a microcomputer.
 9. The imagedisplaying method according to claim 6 further comprising the step ofgenerating the image data of the view of the virtual camera remotecontrolled in response to the detected movement of the image input unitby a camera device located in a real space and remote controlled. 10.The image displaying method according to claim 6, wherein the step ofprocessing the image data to detect the movement of the image displayunit includes calculating a change in the barycenter of the input imagebetween two consecutive frames produced by the image input unit.